{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 垃圾邮件分类\n",
    "<u>垃圾邮件分类01</u>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "老师在网上下载的公开的英文邮件的数据，拿来做学习和练习<br>\n",
    "数据放在：模块五机器学习\\第一章：机器学习（第九周课程）\\7.垃圾邮件分类\\代码\\datasets\\spam <br>\n",
    "英文邮件的处理和汉字文件的处理原理上有所不同，单操作起来基本是一样的。<br>\n",
    "英文邮件词汇是通过空格来分割的，而汉字的词汇的分割可以通过一个第三方包结巴分词(音)就可以对一篇文章进行分割\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 一，查看了解一下邮件内容"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "SPAM_PATH = os.path.join(\"datasets\", \"spam\")  # 合并出数据存放的路径：当前路径/datasets/spam\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "HAM_DIR = os.path.join(SPAM_PATH, \"easy_ham\")  # 合成正常邮件的路径\n",
    "SPAM_DIR = os.path.join(SPAM_PATH, \"spam\")  # 合并成垃圾邮件的路径\n",
    "ham_filenames = [name for name in sorted(os.listdir(HAM_DIR)) if len(name) > 20] # 获取正常邮件的文件名，并排序\n",
    "spam_filenames = [name for name in sorted(os.listdir(SPAM_DIR)) if len(name) > 20] # 获取垃圾邮件的文件名，并排序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 我们可以使用python的“email”模块解析这些电子邮件（它处理邮件头、编码等）\n",
    "import email\n",
    "import email.policy\n",
    "\n",
    "def load_email(is_spam, filename, spam_path=SPAM_PATH):   # 三个参数：是否垃圾邮件，文件名，当前默认路径\n",
    "    directory = \"spam\" if is_spam else \"easy_ham\"   # 如果是垃圾邮件，使用路径span,否则使用路径easy_ham\n",
    "    with open(os.path.join(spam_path, directory, filename), \"rb\") as f:  # 以二进制方式打开文件\n",
    "        return email.parser.BytesParser(policy=email.policy.default).parse(f) # 读取文件内容，转换成文字输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Martin A posted:\n",
      "Tassos Papadopoulos, the Greek sculptor behind the plan, judged that the\n",
      " limestone of Mount Kerdylio, 70 miles east of Salonika and not far from the\n",
      " Mount Athos monastic community, was ideal for the patriotic sculpture. \n",
      " \n",
      " As well as Alexander's granite features, 240 ft high and 170 ft wide, a\n",
      " museum, a restored amphitheatre and car park for admiring crowds are\n",
      "planned\n",
      "---------------------\n",
      "So is this mountain limestone or granite?\n",
      "If it's limestone, it'll weather pretty fast.\n",
      "\n",
      "------------------------ Yahoo! Groups Sponsor ---------------------~-->\n",
      "4 DVDs Free +s&p Join Now\n",
      "http://us.click.yahoo.com/pt6YBB/NXiEAA/mG3HAA/7gSolB/TM\n",
      "---------------------------------------------------------------------~->\n",
      "\n",
      "To unsubscribe from this group, send an email to:\n",
      "forteana-unsubscribe@egroups.com\n",
      "\n",
      " \n",
      "\n",
      "Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/\n"
     ]
    }
   ],
   "source": [
    "# 让我们看一个ham示例和一个spam示例，了解数据的外观：\n",
    "ham_emails = [load_email(is_spam=False, filename=name) for name in ham_filenames]   # 转换所以的正常邮件\n",
    "spam_emails = [load_email(is_spam=True, filename=name) for name in spam_filenames]  # 转换所以的垃圾邮件\n",
    "print(ham_emails[1].get_content().strip())   # 查看第二个正常邮件，strip是去除两边空格"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help wanted.  We are a 14 year old fortune 500 company, that is\n",
      "growing at a tremendous rate.  We are looking for individuals who\n",
      "want to work from home.\n",
      "\n",
      "This is an opportunity to make an excellent income.  No experience\n",
      "is required.  We will train you.\n",
      "\n",
      "So if you are looking to be employed from home with a career that has\n",
      "vast opportunities, then go:\n",
      "\n",
      "http://www.basetel.com/wealthnow\n",
      "\n",
      "We are looking for energetic and self motivated people.  If that is you\n",
      "than click on the link and fill out the form, and one of our\n",
      "employement specialist will contact you.\n",
      "\n",
      "To be removed from our link simple go to:\n",
      "\n",
      "http://www.basetel.com/remove.html\n",
      "\n",
      "\n",
      "4139vOLW7-758DoDY1425FRhM1-764SMFc8513fCsLl40\n"
     ]
    }
   ],
   "source": [
    "print(spam_emails[6].get_content().strip())  # 查看第七个垃圾邮件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 电子邮件实际上有很多部分，带有图像和附件（它们可以有自己的附件）。\n",
    "# 定义一个函数查看邮件的各种类型的结构：\n",
    "def get_email_structure(email):\n",
    "    if isinstance(email, str):\n",
    "        return email\n",
    "    payload = email.get_payload()  # 获取邮件组成部分\n",
    "    if isinstance(payload, list):   # 如果是复杂的邮件,附件本身也是邮件，payload会获得list列表\n",
    "        return \"multipart({})\".format(\", \".join([\n",
    "            get_email_structure(sub_email)  # 调取这个函数自己，获取子邮件的结构\n",
    "            for sub_email in payload\n",
    "        ]))\n",
    "    else:\n",
    "        return email.get_content_type()  # 如果没有获得list列表，就说明是简单邮件，直接获取内容类型就可以"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({1: 1, 4: 2, 2: 3, 3: 2})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# counter的作用是计算列表中，各个元素的出现次数\n",
    "from collections import Counter\n",
    "\n",
    "a = [1,4,2,3,2,3,4,2]  \n",
    "b = Counter(a) #求数组中每个数字出现了几次\n",
    "b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<u>垃圾邮件分类02</u>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "def structures_counter(emails):   # 定义一个函数，来读取所有邮件，分析所有这些邮件的结构，并记录合计数目\n",
    "    structures = Counter()\n",
    "    for email in emails:\n",
    "        structure = get_email_structure(email)\n",
    "        structures[structure] += 1\n",
    "    return structures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('text/plain', 2408),\n",
       " ('multipart(text/plain, application/pgp-signature)', 66),\n",
       " ('multipart(text/plain, text/html)', 8),\n",
       " ('multipart(text/plain, text/plain)', 4),\n",
       " ('multipart(text/plain)', 3),\n",
       " ('multipart(text/plain, application/octet-stream)', 2),\n",
       " ('multipart(text/plain, text/enriched)', 1),\n",
       " ('multipart(text/plain, application/ms-tnef, text/plain)', 1),\n",
       " ('multipart(multipart(text/plain, text/plain, text/plain), application/pgp-signature)',\n",
       "  1),\n",
       " ('multipart(text/plain, video/mng)', 1),\n",
       " ('multipart(text/plain, multipart(text/plain))', 1),\n",
       " ('multipart(text/plain, application/x-pkcs7-signature)', 1),\n",
       " ('multipart(text/plain, multipart(text/plain, text/plain), text/rfc822-headers)',\n",
       "  1),\n",
       " ('multipart(text/plain, multipart(text/plain, text/plain), multipart(multipart(text/plain, application/x-pkcs7-signature)))',\n",
       "  1),\n",
       " ('multipart(text/plain, application/x-java-applet)', 1)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "structures_counter(ham_emails).most_common()   # 查看正常邮件的统计结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('text/plain', 218),\n",
       " ('text/html', 183),\n",
       " ('multipart(text/plain, text/html)', 45),\n",
       " ('multipart(text/html)', 20),\n",
       " ('multipart(text/plain)', 19),\n",
       " ('multipart(multipart(text/html))', 5),\n",
       " ('multipart(text/plain, image/jpeg)', 3),\n",
       " ('multipart(text/html, application/octet-stream)', 2),\n",
       " ('multipart(text/plain, application/octet-stream)', 1),\n",
       " ('multipart(text/html, text/plain)', 1),\n",
       " ('multipart(multipart(text/html), application/octet-stream, image/jpeg)', 1),\n",
       " ('multipart(multipart(text/plain, text/html), image/gif)', 1),\n",
       " ('multipart/alternative', 1)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "structures_counter(spam_emails).most_common()  # 查看异常邮件的统计结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "正常邮件更多的是纯文本，而垃圾邮件有相当多的HTML。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Return-Path : <12a1mailbot1@web.de>\n",
      "Delivered-To : zzzz@localhost.spamassassin.taint.org\n",
      "Received : from localhost (localhost [127.0.0.1])\tby phobos.labs.spamassassin.taint.org (Postfix) with ESMTP id 136B943C32\tfor <zzzz@localhost>; Thu, 22 Aug 2002 08:17:21 -0400 (EDT)\n",
      "Received : from mail.webnote.net [193.120.211.219]\tby localhost with POP3 (fetchmail-5.9.0)\tfor zzzz@localhost (single-drop); Thu, 22 Aug 2002 13:17:21 +0100 (IST)\n",
      "Received : from dd_it7 ([210.97.77.167])\tby webnote.net (8.9.3/8.9.3) with ESMTP id NAA04623\tfor <zzzz@spamassassin.taint.org>; Thu, 22 Aug 2002 13:09:41 +0100\n",
      "From : 12a1mailbot1@web.de\n",
      "Received : from r-smtp.korea.com - 203.122.2.197 by dd_it7  with Microsoft SMTPSVC(5.5.1775.675.6);\t Sat, 24 Aug 2002 09:42:10 +0900\n",
      "To : dcek1a1@netsgo.com\n",
      "Subject : Life Insurance - Why Pay More?\n",
      "Date : Wed, 21 Aug 2002 20:31:57 -1600\n",
      "MIME-Version : 1.0\n",
      "Message-ID : <0103c1042001882DD_IT7@dd_it7>\n",
      "Content-Type : text/html; charset=\"iso-8859-1\"\n",
      "Content-Transfer-Encoding : quoted-printable\n"
     ]
    }
   ],
   "source": [
    "# 查看某邮件头部信息\n",
    "for header, value in spam_emails[0].items():\n",
    "    print(header,\":\",value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "里面可能有很多有用的信息，比如发件人的电子邮件地址（12a1mailbot1@web.de看起来很可疑）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Life Insurance - Why Pay More?'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看“主题”标题：\n",
    "spam_emails[0][\"Subject\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二，拆分训练集和测试集合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 拆分测试集和训练集\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X = np.array(ham_emails + spam_emails)   # 数据是所以的正常邮件+垃圾邮件\n",
    "y = np.array([0] * len(ham_emails) + [1] * len(spam_emails))  # 正常邮件标签是0，垃圾邮件标签是1\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # 按照8：2的比例拆分训练集和测试集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 三，数据处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* head部分基本上对于判断是否是垃圾邮件作用不大，可以删掉\n",
    "* html格式会影响训练效果，需要一个函数来将html转换为纯文本\n",
    "* 可以使用[Beautifulsoup]库，或者正则表达式来实现。\n",
    "* 下面的函数使用正则表达式，首先删除`<head>`部分，然后将所有`<a>`标记转换为单词hyperlink，然后去掉所有html标记，只留下纯文本。\n",
    "* 为了可读性，它还用一个换行符替换多个换行符，最后它取消了HTML实体（例如`&gt；`或`&nbsp；`）\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re   # 正则表达式\n",
    "from html import unescape\n",
    "\n",
    "def html_to_plain_text(html):   # 删除head，a标签，空格等\n",
    "    text = re.sub('<head.*?>.*?</head>', '', html, flags=re.M | re.S | re.I)  # 删除head\n",
    "    text = re.sub('<a\\s.*?>', ' HYPERLINK ', text, flags=re.M | re.S | re.I)  # 将a标签转换成文本\n",
    "    text = re.sub('<.*?>', '', text, flags=re.M | re.S)   #  \n",
    "    text = re.sub(r'(\\s*\\n)+', '\\n', text, flags=re.M | re.S)  # \n",
    "    return unescape(text)   # 返回调整后的文本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<HTML><HEAD><TITLE></TITLE><META http-equiv=\"Content-Type\" content=\"text/html; charset=windows-1252\"><STYLE>A:link {TEX-DECORATION: none}A:active {TEXT-DECORATION: none}A:visited {TEXT-DECORATION: none}A:hover {COLOR: #0033ff; TEXT-DECORATION: underline}</STYLE><META content=\"MSHTML 6.00.2713.1100\" name=\"GENERATOR\"></HEAD>\n",
      "<BODY text=\"#000000\" vLink=\"#0033ff\" link=\"#0033ff\" bgColor=\"#CCCC99\"><TABLE borderColor=\"#660000\" cellSpacing=\"0\" cellPadding=\"0\" border=\"0\" width=\"100%\"><TR><TD bgColor=\"#CCCC99\" valign=\"top\" colspan=\"2\" height=\"27\">\n",
      "<font size=\"6\" face=\"Arial, Helvetica, sans-serif\" color=\"#660000\">\n",
      "<b>OTC</b></font></TD></TR><TR><TD height=\"2\" bgcolor=\"#6a694f\">\n",
      "<font size=\"5\" face=\"Times New Roman, Times, serif\" color=\"#FFFFFF\">\n",
      "<b>&nbsp;Newsletter</b></font></TD><TD height=\"2\" bgcolor=\"#6a694f\"><div align=\"right\"><font color=\"#FFFFFF\">\n",
      "<b>Discover Tomorrow's Winners&nbsp;</b></font></div></TD></TR><TR><TD height=\"25\" colspan=\"2\" bgcolor=\"#CCCC99\"><table width=\"100%\" border=\"0\"  ...\n"
     ]
    }
   ],
   "source": [
    "html_spam_emails = [email for email in X_train[y_train==1]   # 代入所以的垃圾邮件中\n",
    "                    if get_email_structure(email) == \"text/html\"]   # 邮件结构中包含\"text/html\"的\n",
    "sample_html_spam = html_spam_emails[7]       # 拿其中第8个垃圾邮件看看效果\n",
    "print(sample_html_spam.get_content().strip()[:1000], \"...\")  # 取前1000个字符，后面的用..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "OTC\n",
      " Newsletter\n",
      "Discover Tomorrow's Winners \n",
      "For Immediate Release\n",
      "Cal-Bay (Stock Symbol: CBYI)\n",
      "Watch for analyst \"Strong Buy Recommendations\" and several advisory newsletters picking CBYI.  CBYI has filed to be traded on the OTCBB, share prices historically INCREASE when companies get listed on this larger trading exchange. CBYI is trading around 25 cents and should skyrocket to $2.66 - $3.25 a share in the near future.\n",
      "Put CBYI on your watch list, acquire a position TODAY.\n",
      "REASONS TO INVEST IN CBYI\n",
      "A profitable company and is on track to beat ALL earnings estimates!\n",
      "One of the FASTEST growing distributors in environmental & safety equipment instruments.\n",
      "Excellent management team, several EXCLUSIVE contracts.  IMPRESSIVE client list including the U.S. Air Force, Anheuser-Busch, Chevron Refining and Mitsubishi Heavy Industries, GE-Energy & Environmental Research.\n",
      "RAPIDLY GROWING INDUSTRY\n",
      "Industry revenues exceed $900 million, estimates indicate that there could be as much as $25 billi ...\n"
     ]
    }
   ],
   "source": [
    "print(html_to_plain_text(sample_html_spam.get_content())[:1000], \"...\")  # 删除head，a标签，空格等之后再看看效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 把上面的整合一下，编写一个函数，它以电子邮件为输入，无论其格式是什么，都以纯文本形式返回其内容，\n",
    "def email_to_text(email):\n",
    "    html = None\n",
    "    for part in email.walk():\n",
    "        ctype = part.get_content_type()\n",
    "        if not ctype in (\"text/plain\", \"text/html\"):\n",
    "            continue\n",
    "        try:\n",
    "            content = part.get_content()\n",
    "        except: # 解决编码问题\n",
    "            content = str(part.get_payload())\n",
    "        if ctype == \"text/plain\":\n",
    "            return content\n",
    "        else:\n",
    "            html = content\n",
    "    if html:\n",
    "        return html_to_plain_text(html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "OTC\n",
      " Newsletter\n",
      "Discover Tomorrow's Winners \n",
      "For Immediate Release\n",
      "Cal-Bay (Stock Symbol: CBYI)\n",
      "Watch for analyst \"Strong Buy Recommendations\" and several advisory newsletters picking CBYI.  CBYI has filed to be traded on the OTCBB, share prices historically INCREASE when companies get listed on this larger trading exchange. CBYI is trading around 25 cents and should skyrocket to $2.66 - $3.25 a share in the near future.\n",
      "Put CBYI on your watch list, acquire a position TODAY.\n",
      "REASONS TO INVEST IN CBYI\n",
      "A profitable company and is on track to beat ALL earnings estimates!\n",
      "One of the FASTEST growing distributors in environmental & safety equipment instruments.\n",
      "Excellent management team, several EXCLUSIVE contracts.  IMPRESSIVE client list including the U.S. Air Force, Anheuser-Busch, Chevron Refining and Mitsubishi Heavy Industries, GE-Energy & Environmental Research.\n",
      "RAPIDLY GROWING INDUSTRY\n",
      "Industry revenues exceed $900 million, estimates indicate that there could be as much as $25 billi ...\n"
     ]
    }
   ],
   "source": [
    "print(email_to_text(sample_html_spam)[:1000],\"...\")  # 验证一下这个函数，输出前1000字符"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "接下来对文本进行 拆分统计<br>\n",
    "需要安装两个包<br>\n",
    " 装自然语言工具包（[nltk]（http://www.nltk.org/）<br>\n",
    " &nbsp;&nbsp;pip install nltk<br>\n",
    "\n",
    " 用“url”这个词来替换url的方法的包 （防止机器把过多的时间放在解析url链接上）<br>\n",
    " &nbsp;&nbsp;pip install urlextract<br>\n",
    " \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <u>垃圾邮件分类03</u>\n",
    " \n",
    "## 四，定义转换（计数）器\n",
    "利用上面两个包，将所有处理整合到一个转换器中，我们将使用它将电子邮件转换为文字计数器。<br>\n",
    "注意，因为数据集（主要）是英文的，我们使用python的'split（）'方法将句子拆分为单词，该方法使用空格作为单词边界。<br>\n",
    "但要注意，汉语和日语脚本通常不在单词之间使用空格，中文可以使用结巴分词来进行拆分"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\00-app\\python\\python38\\lib\\site-packages\\sklearn\\feature_extraction\\text.py:17: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working\n",
      "  from collections import Mapping, defaultdict\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computations => comput\n",
      "Computation => comput\n",
      "Computing => comput\n",
      "Computed => comput\n",
      "Compute => comput\n",
      "Compulsive => compuls\n"
     ]
    }
   ],
   "source": [
    "# 拿个例子看一下 nltk 的作用，将一个词的不同形态 拆解成同一个词/词干。\n",
    "import nltk\n",
    "from urlextract import URLExtract\n",
    "\n",
    "try:\n",
    "    import nltk\n",
    "\n",
    "    stemmer = nltk.PorterStemmer()\n",
    "    for word in (\"Computations\", \"Computation\", \"Computing\", \"Computed\", \"Compute\", \"Compulsive\"):\n",
    "        print(word, \"=>\", stemmer.stem(word))\n",
    "except ImportError:\n",
    "    print(\"Error: stemming requires the NLTK module.\")\n",
    "    stemmer = None\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的例子我们可以看出，把计算相关的词汇的变成了comput这个样子；Compulsive没有计算的意思，没有分解成comput"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator, TransformerMixin # 导入相关模块\n",
    "\n",
    "class EmailToWordCounterTransformer(BaseEstimator, TransformerMixin):  # 继承两个基类\n",
    "    def __init__(self, strip_headers=True, lower_case=True, remove_punctuation=True,\n",
    "                 replace_urls=True, replace_numbers=True,  stemming=True):   # 初始化，默认都是True\n",
    "        self.strip_headers = strip_headers  # 是否除去头部\n",
    "        self.lower_case = lower_case   # 转换为小写\n",
    "        self.remove_punctuation = remove_punctuation  # 去掉标点\n",
    "        self.replace_urls = replace_urls  # 更换urls为文本\n",
    "        self.replace_numbers = replace_numbers  # 置换数字\n",
    "        self.stemming = stemming  \n",
    "    def fit(self, X, y=None):  # 训练这里，什么都不做，这个函数主要做转换\n",
    "        return self\n",
    "    def transform(self, X, y=None):  # 传入X就是emails(很多邮件)\n",
    "        X_transformed = []\n",
    "        for email in X:  # 遍历每一个邮件\n",
    "            text = email_to_text(email) or \"\"   # 利用之前写好的函数将邮件转换成纯文本\n",
    "            if self.lower_case:  # 转换小写\n",
    "                text = text.lower()\n",
    "            if self.replace_urls:  # 转换url为\"url\"这个字符\n",
    "                extractor = URLExtract()\n",
    "                urls = list(set(extractor.find_urls(text)))  # 整片文章中找url,返回url列表\n",
    "                urls.sort(key=lambda url: len(url), reverse=True)  # 进行一个排序，（不知道为什么）\n",
    "                for url in urls:    # 替换url 为 文本：‘URL’\n",
    "                    text = text.replace(url, \" URL \")\n",
    "            if self.replace_numbers:   # 替换数字为字符'NUMBER'\n",
    "                text = re.sub(r'\\d+(?:\\.\\d*(?:[eE]\\d+))?', 'NUMBER', text)\n",
    "            if self.remove_punctuation:   # 删除标点符号\n",
    "                text = re.sub(r'\\W+', ' ', text, flags=re.M)\n",
    "            word_counts = Counter(text.split())    # 将邮件文本拆分成单词，并计算各个单词的出现次数\n",
    "            if self.stemming and stemmer is not None:  # 如果要求按词根计数，并且有词根，就安装词根计数\n",
    "                stemmed_word_counts = Counter()\n",
    "                for word, count in word_counts.items():\n",
    "                    stemmed_word = stemmer.stem(word)   # 词根\n",
    "                    stemmed_word_counts[stemmed_word] += count  # 词根数\n",
    "                word_counts = stemmed_word_counts\n",
    "            X_transformed.append(word_counts)  # 获取词根计数\n",
    "        return np.array(X_transformed)\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<u>垃圾邮件分类04</u>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([Counter({'chuck': 1, 'murcko': 1, 'wrote': 1, 'stuff': 1, 'yawn': 1, 'r': 1}),\n",
       "       Counter({'the': 11, 'of': 9, 'and': 8, 'all': 3, 'christian': 3, 'to': 3, 'by': 3, 'jefferson': 2, 'i': 2, 'have': 2, 'superstit': 2, 'one': 2, 'on': 2, 'been': 2, 'ha': 2, 'half': 2, 'rogueri': 2, 'teach': 2, 'jesu': 2, 'some': 1, 'interest': 1, 'quot': 1, 'url': 1, 'thoma': 1, 'examin': 1, 'known': 1, 'word': 1, 'do': 1, 'not': 1, 'find': 1, 'in': 1, 'our': 1, 'particular': 1, 'redeem': 1, 'featur': 1, 'they': 1, 'are': 1, 'alik': 1, 'found': 1, 'fabl': 1, 'mytholog': 1, 'million': 1, 'innoc': 1, 'men': 1, 'women': 1, 'children': 1, 'sinc': 1, 'introduct': 1, 'burnt': 1, 'tortur': 1, 'fine': 1, 'imprison': 1, 'what': 1, 'effect': 1, 'thi': 1, 'coercion': 1, 'make': 1, 'world': 1, 'fool': 1, 'other': 1, 'hypocrit': 1, 'support': 1, 'error': 1, 'over': 1, 'earth': 1, 'six': 1, 'histor': 1, 'american': 1, 'john': 1, 'e': 1, 'remsburg': 1, 'letter': 1, 'william': 1, 'short': 1, 'again': 1, 'becom': 1, 'most': 1, 'pervert': 1, 'system': 1, 'that': 1, 'ever': 1, 'shone': 1, 'man': 1, 'absurd': 1, 'untruth': 1, 'were': 1, 'perpetr': 1, 'upon': 1, 'a': 1, 'larg': 1, 'band': 1, 'dupe': 1, 'import': 1, 'led': 1, 'paul': 1, 'first': 1, 'great': 1, 'corrupt': 1}),\n",
       "       Counter({'url': 4, 's': 3, 'group': 3, 'to': 3, 'in': 2, 'forteana': 2, 'martin': 2, 'an': 2, 'and': 2, 'we': 2, 'is': 2, 'yahoo': 2, 'unsubscrib': 2, 'y': 1, 'adamson': 1, 'wrote': 1, 'for': 1, 'altern': 1, 'rather': 1, 'more': 1, 'factual': 1, 'base': 1, 'rundown': 1, 'on': 1, 'hamza': 1, 'career': 1, 'includ': 1, 'hi': 1, 'belief': 1, 'that': 1, 'all': 1, 'non': 1, 'muslim': 1, 'yemen': 1, 'should': 1, 'be': 1, 'murder': 1, 'outright': 1, 'know': 1, 'how': 1, 'unbias': 1, 'memri': 1, 'don': 1, 't': 1, 'html': 1, 'rob': 1, 'sponsor': 1, 'number': 1, 'dvd': 1, 'free': 1, 'p': 1, 'join': 1, 'now': 1, 'from': 1, 'thi': 1, 'send': 1, 'email': 1, 'egroup': 1, 'com': 1, 'your': 1, 'use': 1, 'of': 1, 'subject': 1})],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 在一些邮件上 测试 上面写好的转换器\n",
    "X_few = X_train[:3]  # 拿头三个邮件来测试\n",
    "X_few_wordcounts = EmailToWordCounterTransformer().fit_transform(X_few)\n",
    "X_few_wordcounts # 结果中可以看到三个邮件中，每个邮件中单词的出现次数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 有了单词计数，我们需要把它们转换成向量。为此，我们将构建另一个转换器，\n",
    "## 五，构建一个转换器：常用词汇列表/出现次数\n",
    "* 其“fit（）”方法将构建词汇表（最常用单词的有序列表），\n",
    "* 其“transform（）”方法将使用词汇表将单词计数转换为向量--稀疏矩阵"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.sparse import csr_matrix\n",
    "\n",
    "class WordCounterToVectorTransformer(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self, vocabulary_size = 1000):\n",
    "        self.vocabulary_size = vocabulary_size  # 初始化词汇量，默认指定1000个\n",
    "    def fit(self, X, y = None):  # 训练数据\n",
    "        total_count = Counter()  \n",
    "        for word_count in X:   # 读取数据集中每一个单词\n",
    "            for word, count in word_count.items():   # 记录单词、数量\n",
    "                total_count[word] += min(count, 10)    # 最大计数设置为10 哪怕是1000/10000也限制为10\n",
    "        most_common = total_count.most_common()[:self.vocabulary_size]  # 利用Counter方法中选取出现频率最高的方法，选取前1000个\n",
    "        self.most_common_ = most_common  # 把最常用的词汇表付给变量\n",
    "        self.vocabulary_ = {word: index + 1 for index, (word, count) in enumerate(most_common)} # 将最常用词汇表付给变量，生成字典，序列，单词，数量\n",
    "        return self\n",
    "    def transform(self, X, y = None): # 主要作用是某邮件中，最常见单词的出现次数统计，最后形成稀疏矩阵\n",
    "        rows = []  # 行数\n",
    "        cols = []  # 列数\n",
    "        data = []  # （行号，列号）对应的值\n",
    "        for row, word_count in enumerate(X):  # 每一行row就是一封邮件，对邮件遍历\n",
    "            for word, count in word_count.items():  # 对邮件里的单词项目遍历\n",
    "                rows.append(row)   # 训练集中 实例个数\n",
    "                cols.append(self.vocabulary_.get(word, 0))  # 取得该单词在词汇表中的索引位置，未出现在词汇表中则用0代替\n",
    "                data.append(count)   # 该单词出现次数\n",
    "                \n",
    "        # 输出稀疏矩阵 +1因为第一列要显示未出现在词汇表中的单词统计数\n",
    "        return csr_matrix((data, (rows, cols)), shape=(len(X), self.vocabulary_size + 1)) # 行数是邮件数，列数是常见单词数量+1 ，第0列为非常见单词计数列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 1],\n",
       "       [0, 1, 1],\n",
       "       [1, 1, 0]], dtype=int32)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ## csr这个稀疏矩阵使用方法例子\n",
    "# from scipy.sparse import *\n",
    " \n",
    "# row =  [0,0,0,1,1,1,2,2,2]  #行指标\n",
    "# col =  [0,1,2,0,1,2,0,1,2]  #列指标\n",
    "# data = [1,0,1,0,1,1,1,1,0]  #在行指标列指标下的数字\n",
    "# team = csr_matrix((data,(row,col)),shape=(3,3))   # 塑形成3行3列\n",
    "# print(team)   # 稀疏矩阵方式输出\n",
    "# print(team.todense())   # 以矩阵方式输出\n",
    "# team.toarray()   # 以矩阵方式输出\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<u>垃圾邮件分类05</u>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 11)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 实际跑一下上面写好的函数\n",
    "vocab_transformer = WordCounterToVectorTransformer(vocabulary_size=10)  # 取10个常用单词做练习\n",
    "X_few_vectors = vocab_transformer.fit_transform(X_few_wordcounts)  # 获取数据，X_few_wordcounts在之前定义是3个邮件\n",
    "X_few_vectors.shape  # 查看一下，是3行（每行一个邮件），11列（查看前11个常用词汇，在该邮件中出现次数）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 6,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0],\n",
       "       [99, 11,  9,  8,  3,  1,  3,  1,  3,  2,  3],\n",
       "       [67,  0,  1,  2,  3,  4,  1,  2,  0,  1,  0]], dtype=int32)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_few_vectors.toarray()  # 查看生成结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面输出的意思：\n",
    "* 每行代表对一个邮件的统计\n",
    "* 第一列表示，不属于常用单词的计数综合\n",
    "* 第二列表示，出现频率排第一的单词出现次数，依次类推\n",
    "* 下面👇的输出可以看排第1，第2，第3。。。的单词是是啥"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'the': 1,\n",
       " 'of': 2,\n",
       " 'and': 3,\n",
       " 'to': 4,\n",
       " 'url': 5,\n",
       " 'all': 6,\n",
       " 'in': 7,\n",
       " 'christian': 8,\n",
       " 'on': 9,\n",
       " 'by': 10}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab_transformer.vocabulary_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 六，准备训练我们的第一个垃圾邮件分类器！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 首先，让我们转换整个数据集，训练模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 建立一个pipeline,把之前写的函数合并起来\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "preprocess_pipeline = Pipeline([\n",
    "    (\"email_to_wordcount\", EmailToWordCounterTransformer()),\n",
    "    (\"wordcount_to_vector\", WordCounterToVectorTransformer()),\n",
    "])\n",
    "# 拿整个训练集来训练模型\n",
    "X_train_transformed = preprocess_pipeline.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 然后，查看一下模型精度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  ................................................................\n",
      "[CV] .................................. , score=0.98125, total=   0.5s\n",
      "[CV]  ................................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.7s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .................................. , score=0.98375, total=   0.3s\n",
      "[CV]  ................................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    1.1s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .................................. , score=0.99125, total=   0.8s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:    2.0s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.9854166666666666"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression   # 逻辑回归分类器\n",
    "from sklearn.model_selection import cross_val_score   # 交叉认证计算精度\n",
    "\n",
    "log_clf = LogisticRegression(solver=\"liblinear\", random_state=42) # 采用逻辑回归分类器\n",
    "score = cross_val_score(log_clf, X_train_transformed, y_train, cv=3, verbose=3)   # 训练集上交叉认证评估一下这个分类器，看看精度是多少\n",
    "score.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 👆得到分数超过98.7%，\n",
    "* 也可以尝试多个模型，选择最好的模型，并使用交叉验证对它们进行微调。<br>\n",
    "\n",
    "### 拿训练好的模型测试集上获得结果，最后计算测试集下的精度/召回率："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "精度: 96.88%\n",
      "召回: 97.89%\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score, recall_score\n",
    "\n",
    "X_test_transformed = preprocess_pipeline.transform(X_test)  # 代入测试集，数据整形\n",
    "\n",
    "log_clf = LogisticRegression(solver=\"liblinear\", random_state=42)   # 使用逻辑回归模型\n",
    "log_clf.fit(X_train_transformed, y_train)   # 对测试集进行训练\n",
    "\n",
    "y_pred = log_clf.predict(X_test_transformed)  # 对测试集进行预测\n",
    "\n",
    "print(\"精度: {:.2f}%\".format(100 * precision_score(y_test, y_pred)))  # 将实际测试标签和预测的测试结果，拿来算出精度\n",
    "print(\"召回: {:.2f}%\".format(100 * recall_score(y_test, y_pred)))  # 将实际测试标签和预测的测试结果，拿来算出召回率"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "1. 加载数据并纵了解观数据大局\n",
    "2. 获取邮件的组成结构\n",
    "3. 对结构类型进行分析 发现垃圾邮件大多有HTML结构\n",
    "4. 数据清洗，定义email对象中的HTML转换称纯文本方法\n",
    "5. 对数据集拆分成训练集和测试集\n",
    "6. 数据处理转换，对邮件的文本内容进行分词处理，通过nltk进行词干提取，对邮件出现的词汇进行计数统计，对所有邮件统计出了一个常用词汇表\n",
    "7. 通过词汇表和邮件单词计数统计，将单词计数转化成向量矩阵\n",
    "8. 把数据清洗和数据处理封装成两个转换器\n",
    "9. 通过流水线来自动化处理数据\n",
    "10. 使用逻辑回归线性分类器进行模型训练\n",
    "11. 使用交叉验证进行微调\n",
    "12. 在测试集上得到精度/召回率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
